Composite image generating apparatus, composite image generating method, and storage medium

ABSTRACT

A plurality of items of shot image data (Gn) obtained by temporally continuous shooting are analyzed. Marking data ( 24 Mw) indicating that replaced graphic data ( 24 Gw) is to be combined is added to the image data corresponding to actor P and the resulting data is displayed. When a preset gesture (motion) is detected, marking data ( 14 Mu) indicating that replaced graphic data ( 14 Gu) is to be combined is added to the image data corresponding to actor K and the resulting data is displayed. After shooting, the individual items of image data to which marking data ( 14 Mu,  15 Ma,  15 Mb, and  24 Mw) have been added are replaced with replaced graphic data ( 24 Gw), replaced graphic data ( 14 Gu), replaced graphic data ( 15 Ga), and replaced graphic data ( 15 Gb), respectively. Replaced graphic data ( 24 Gw) and replaced graphic data ( 14 Gu) are created as moving images which capture the motions of actors K and P.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is based upon and claims the benefit of priority fromprior Japanese Patent Applications No. 2007-222595, filed Aug. 29, 2007;and No. 2008-197627, filed Jul. 31, 2008, the entire contents of both ofwhich are incorporated herein by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

This invention relates to a composite image generating apparatus, acomposite image generating method, and a storage medium which combine ashot image with another image.

2. Description of the Related Art

A conventional image output apparatus extracts a human figure from animage shot with a digital camera, determines the posture of theextracted human figure, and combines a character image with the humanfigure according to the determined posture, thereby displaying theresulting image.

BRIEF SUMMARY OF THE INVENTION

It is an object of the invention to provide a composite image generatingapparatus, a composite image generating method, and a program productwhich determine the changing part from a plurality of items of imagedata and produce image data corresponding to the changing part.

The foregoing object is achieved by providing a composite imagegenerating apparatus comprising: a first storage unit configured tostore a motion to be combined and first composite image data in such amanner that the motion and the first composite image data correspond toeach other; an input unit configured to input a plurality of items ofimage data; a first determining unit configured to determine whether ornot the movement of a part changing over a plurality of items of imagedata input by the input unit is almost the same as the motion stored inthe first storage unit; and a first composite unit configured to readthe first composite image data stored in the first storage unit so as tocorrespond to the motion and combine the first composite data with theimage data having the changing part, if the first determining unitdetermines that the movement is almost the same as the motion.

The foregoing object is further achieved by providing a composite imagegenerating method comprising: an input step of inputting a plurality ofitems of image data; a determination step of determining whether or notthe movement of a part changing over a plurality of items of image datainput in the input step is almost the same as a motion set so as tocorrespond to composite image data; and a composite step of, if it isdetermined in the determination step that the movement is almost thesame as the motion, combining the composite image data set so as tocorrespond to the motion with image data having the changing part.

The foregoing object is further achieved by providing A computerreadable medium for storing a program product for use with imagegenerating apparatus including an image input unit, the program productcomprising:

first computer readable program means for inputting a plurality of itemsof image data by the image input unit, second computer readable programmeans for determining whether or not the movement of a part changingover a plurality of items of image data input by the first computerreadable program means is almost the same as a motion set so as tocorrespond to composite image data, and third computer readable programmeans for, if the second computer readable program means determines thatthe movement is almost the same as the motion, combining the compositeimage data set so as to correspond to the motion with the image datahaving the changing part.

Additional objects and advantages of the invention will be set forth inthe description which follows, and in part will be obvious from thedescription, or may be learned by practice of the invention. The objectsand advantages of the invention may be realized and obtained by means ofthe instrumentalities and combinations particularly pointed outhereinafter.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING

The accompanying drawings, which are incorporated in and constitute apart of the specification, illustrate embodiments of the invention, andtogether with the general description given above and the detaileddescription of the embodiments given below, serve to explain theprinciples of the invention.

FIG. 1 shows a state where a moving image is shot in a first embodimentof the invention;

FIG. 2A shows a state where an image is displayed on a display unit 11,and FIG. 2B shows a composite image displayed on the display unit 11after shooting;

FIG. 3 is a block diagram showing a circuit configuration of a digitalcamera 10 of the embodiment;

FIG. 4 shows contents previously stored in a changing motion data memory14 of the embodiment;

FIG. 5 shows contents previously stored in a special effect graphic datamemory 23 of the embodiment;

FIG. 6 shows contents previously stored in a replaced graphic datamemory (for objects) 15 of the embodiment;

FIG. 7 shows contents previously stored in a replaced graphic datamemory (for actors) 24 of the embodiment;

FIG. 8 is a block diagram showing a configuration of the electroniccircuit of a server 30 of the embodiment;

FIG. 9 is a flowchart to help explain all the steps of a composite imageoutput process of the embodiment;

FIG. 10 is a sub-flowchart to help explain the process in step SA ofFIG. 9;

FIG. 11 is a sub-flowchart to help explain the process in step SB ofFIG. 9;

FIG. 12 is a sub-flowchart to help explain the process in step SC ofFIG. 9;

FIG. 13 is a sub-flowchart to help explain the process in step SD ofFIG. 9;

FIG. 14 is a sub-flowchart to help explain the process in step SE ofFIG. 9;

FIG. 15 is a sub-flowchart to help explain the process in step SF ofFIG. 9;

FIG. 16 is a sub-flowchart to help explain the process in step SG ofFIG. 9;

FIG. 17 is a sub-flowchart to help explain the process in step SH ofFIG. 9;

FIG. 18A shows shot image data G1, FIG. 18B shows shot image data (addedmarking data) Gm1, and FIG. 18C shows composite image data GG1 in theembodiment;

FIG. 19A shows shot image data G2, FIG. 19B shows shot image data (addedmarking data) Gm2, and FIG. 19C shows composite image data GG2 in theembodiment;

FIG. 20A shows shot image data G3, FIG. 20B shows shot image data (addedmarking data) Gm3, and FIG. 20C shows composite image data GG3 in theembodiment;

FIG. 21A shows shot image data G4, FIG. 21B shows shot image data (addedmarking data) Gm4, and FIG. 21C shows composite image data GG4 in theembodiment;

FIG. 22A shows shot image data G5, FIG. 22B shows shot image data (addedmarking data) Gm5, and FIG. 22C shows composite image data GG5 in theembodiment;

FIG. 23A shows shot image data G6, FIG. 23B shows shot image data (addedmarking data) Gm6, and FIG. 23C shows composite image data GG6 in theembodiment;

FIG. 24A shows shot image data G7, FIG. 24B shows shot image data (addedmarking data) Gm7, and FIG. 24C shows composite image data GG7 in theembodiment;

FIG. 25A shows shot image data G8, FIG. 25B shows shot image data (addedmarking data) Gm8, and FIG. 25C shows composite image data GG8 in theembodiment; and

FIG. 26A shows shot image data G9, FIG. 26B shows shot image data (addedmarking data) Gm9, and FIG. 26C shows composite image data GG9 in theembodiment.

DETAILED DESCRIPTION OF THE INVENTION

FIG. 1 shows a state where shooting is performed with a digital camera10 according to an embodiment of the invention. FIG. 2A shows a statewhere an image is displayed on a display unit 11. FIG. 2B shows acomposite image displayed on the display unit 11 after shooting.

The digital camera 10 has a display unit 11, an input unit 12, and anaudio output unit (a speaker) 13 on the back of the digital camera 10.Shot image data Gn obtained by operating the input unit 12 is displayedin real time on the display unit 11.

A scene where actor P knocks down object A and knocks off incomingobject B and actor K makes a previously registered gesture is shot inthe form of a video and recorded.

A plurality of items of shot image data Gn obtained by temporallycontinuous shooting are displayed on the display unit 11 in real time asshown in FIG. 2A. At the same time, a plurality of items of shot imagedata are analyzed. Marking data 24Mw indicating that replaced graphicdata 24Gw is to be combined is added to the image data corresponding toactor P and the resulting data is displayed. When a preset gesture isdetected, marking data 14Mu indicating that replaced graphic data 14Guis to be combined is added to the image data corresponding to actor Kand the resulting data is displayed.

In addition, marking data (before contact) 15Ma indicating that replacedgraphic data (before contact) 15Ga is to be combined is added to theimage data corresponding to object A, and the resulting data isdisplayed. Marking data (before contact) 15Mb indicating that replacedgraphic data (before contact) 15Gb is to be combined is added to objectimage data B and the resulting data is displayed.

Furthermore, when shooting is performed, the items of audio datapreviously stored so as to correspond to the individual items of markingdata are read and output from the audio output unit 13.

In the explanation below, marked shot images Gm refers to what isobtained by adding marking data 24Mw, marking data 14Mu, marking data15Ma, and marking data 15Mb to shot moving images.

After shooting, the individual items of image data (target image data)to which marking data 24Mw, 14Mu, 15Ma, and 15Mb have been added in theshot image data (added marking data) Gm are replaced with replacedgraphic data 24Gw, replaced graphic data 14Gu, replaced graphic data15Ga, and replaced graphic data 15Gb, respectively, as shown in FIG. 2B.Replaced graphic data 24Gw and replaced graphic data 14Gu are created asmoving images which capture the motions of actors K and P. These arecombined with background image data BG, thereby creating compositemoving image data GG.

When detecting by image analysis that the target image datacorresponding to object A and the target image data corresponding toobject B come into contact with the target image data corresponding toactor P and actor K, the digital camera 10 replaces not only markingdata (before contact) 15Ma and marking data (before contact) 15Mb withmarking data (after contact) 15Ma′ and marking data (after contact)15Mb′, respectively, but also replaced graphic data (before contact)15Ga and replaced graphic data (before contact) 15Gb with replacedgraphic data (after contact) 15Ga′ and replaced graphic data (aftercontact) 15Gb′, respectively, thereby producing composite data. Thedetails will be described later.

FIG. 3 is a block diagram showing a configuration of the electroniccircuit of the digital camera 10.

The digital camera 10 includes a CPU 16, and a computer.

The CPU 16 controls the operation of each section of the circuitaccording to a system program previously stored in a memory 17, a cameracontrol program read from an external recording medium 18, such as amemory card, into the memory 17 via a recording medium read unit 19,such as a card slot, or a camera control program read from a server 30on a network (N) into the memory 17 via a communication control unit 20.

The CPU 16 includes a processing unit which mainly performs thefollowing functions:

-   -   An extracting unit 161 which recognizes actor K making a        specific motion over a plurality of items of shot image data Gn        continuous in time as a moving object and which further        recognizes object A, object B, and actor P by image analysis and        continuously extracts data as target image data.    -   A first determining unit 162 which recognizes the characterizing        portion and shape of the image data and determines whether the        target image data has been registered.    -   A first marking unit 163 which, if it is determined that the        target image data has been registered, adds marking data to the        target image data in a plurality of items of shot image data in        such a manner that the marking data corresponds to the target        image data and displays the resulting data.    -   A second marking unit 164 which, if it is determined that the        target image data has been registered, determines whether the        target image has been shot at the best angle and, on the basis        of the result of the determination, adds other marking data to        the target image data and displays the resulting data.    -   A second determining unit 165 which determines the positional        relationship between the extracted target image data and other        extracted target image data (more specifically, whether the        extracted target image data makes contact with or overlaps with        other extracted target image data) in a plurality of items of        shot image data.    -   A third marking unit 166 which, if it is determined that the        extracted target image data makes contact with or overlaps with        the other extracted target image data, replaces the marking data        with the other marking data, adds the marking data, and displays        the resulting data.    -   A third determining unit 167 which determines whether the        movement (motion) in a plurality of items of shot image data Gn        in the extracted or marked target image data are almost the same        as the previously registered ones.    -   A fourth marking unit 168 which, if it is determined that the        motion is the previously registered one, adds marking data and        displays the resulting data.    -   A replacing unit 169 which replaces the target image data with        the replaced graphic data corresponding to the marking data for        a plurality of items of shot image data to which marking data        has been added.    -   A moving image generating unit 170 which captures the motion of        target image data and creates moving image data obtained by        moving the target image data three-dimensionally.    -   A composite moving image generating unit 171 which creates        composite moving image data including created moving image data.

The processing programs in the processing units 161 to 171 have beenstored in a program memory 22 and are loaded into the CPU 16 as needed.An image data monitoring process, an image recognition process, acapture process, and a three-dimensional imaging process have beenstored in the image processing program and are loaded as needed.

The program memory 22 has stored a system program which supervises theentire operation of the digital camera 10, a camera control programwhich supervises the photographing operation, a communication controlprogram which supervises the operation of communicating with the server30 on the network N and an external PC (personal computer) as acommunication terminal 40, and an audio output program. These programsare loaded according to a key input signal from the input unit 12, ashot input signal from an image pickup unit 21, or an input signal froman external unit (the server 30, the communication terminal 40) via thecommunication control unit 20.

Connected to the CPU 16 are not only the display unit 11, input unit 12,audio output unit 13, memory 17, recording medium read unit 19, andcommunication control unit 20 but also the image pickup unit 21 whichincludes a solid-state image sensing device (CCD), an imaging opticalsystem, a distance sensor, and an illuminance sensor.

The memory 17 includes a changing motion data memory 14, a specialeffects graphic data memory 23, a replaced graphic data memory (forobjects) 15, a replaced graphic data memory (for actors) 24, a shotimage data memory 25, a shot image data memory (added marking data) 26,a composite image data memory 27, and a working data memory.

In the digital camera 10, a face part 24T of the target image datacorresponding to actor P is stored beforehand into the replaced graphicdata memory (for actors) 24 in such a manner that the face partcorresponds to the replaced graphic data 24Gw.

In addition, target image data 15T is stored beforehand into thereplaced graphic data memory (for objects) 15 in such a manner that thetarget image data corresponds to the replaced graphic data 15Ga.

FIG. 4 shows the contents of data previously stored in the changingmotion data memory 14.

In the changing motion data memory 14, each of a storage area 14 a and astorage area 14 b stores motion data 14P, replaced graphic data 14Gu,marking data 14Mu, and audio data 14Su in such a manner that the motiondata 14P, the replaced graphic data 14Gu, the marking data 14Mu, and theaudio data 14Su correspond to one another.

In the storage area of the marking data 14Mu, marking data 14Mu1, 14Mu2simply representing a plurality of types of replaced graphic data 14Gu1,14Gu2 are stored.

In the storage area of the motion data 14P, motion data 14P1 a to 14P1c, 14P2 a to 14P2 c composed of a series of motions (gestures) forreading replaced graphic data 14Gu1, 14Gu2 are stored.

Here, 3D image generating data representing various motions caused tocorrespond to movement data on the bones of the hands and feet of aperson and on the feature points of the person's face are stored as thereplaced graphic data 14Gu1, 14Gu2.

Moreover, movement data and image data on the bones of the hands andfeet of a person making the motion and on the feature points of theperson's face are stored as the motion data 14P1 a to 14P1 c, 14P2 a to14P2 c.

FIG. 5 shows the contents of data previously stored in the specialeffect graphic data memory 23.

In the special effect graphic data memory 23, each of a storage area 23a and a storage area 23 b stores motion data 23P, replaced graphic data23Gu, marking data 23Mu, and audio data 23Su in such a manner that themotion data 23P, the replaced graphic data 23Gu, the marking data 23Mu,and the audio data 23Su correspond to one another.

In the storage area of the marking data 23Mu, marking data 23Mu1, 23Mu2simply representing a plurality of types of replaced graphic data 23Gu1,23Gu2 are stored.

Stored in the storage area of the motion data 23P are motion data 23P1 ato 23P1 c, 23P2 a to 23P2 c composed of a series of motions (gestures)for combining replaced graphic data 23Gu1, 23Gu2 with specific positionsof the replace graphic data 14Gu1, 14Gu2.

Here, the replaced graphic data 23Gu1, 23Gu2 are stored in the form ofgraphic data or 3D image generating data.

Moreover, movement data and image data on the bones of the hands andfeet of a person making the motion and on the feature points of theperson's face are stored as the motion data 23P1 a to 23P1 c, 23P2 a to23P2 c.

FIG. 6 shows the contents of data previously stored in the replacedgraphic data memory (for objects) 15.

In the replaced graphic data memory (for objects) 15, each of a storagearea 15 a to a storage area 15 stores target image data 15T, replacedgraphic data 15G, marking data (before contact) 15M, marking data (aftercontact) 15M′, and audio data 15S in such a manner that the target imagedata 15T, the replaced graphic data 15G, the marking data (beforecontact) 15M, the marking data (after contact) 15M′, and the audio data15S correspond to one another. In the storage area of the replacedgraphic data 15G, for example, the replaced graphic data 15Ga1 to 15Ga3before contact, those during contact, and those after contact arestored.

In the audio data 15S, audio data 15Sa is set so that its output timingmay be at the time of contact, audio data 15Sb1 is set so that itsoutput timing may be before contact, audio data 15Sb2 is set so that itsoutput timing may be at the time of contact, and audio data 15Sc is setso that its output timing may be after contact.

As for the storage areas of the marking data (before contact) 15M andmarking data (after contact) 15M′, for example, the storage area 15 astores marking data simply representing the replaced graphic data 1Ga1and marking data 15Ma′ simply representing the replaced graphic data15Ga3. In the storage area of the target image data 15T, for example,the storage area 15 a stores the best angle of target image data (e.g.,image data on object A) to be replaced with the replaced graphic data15Ga1 to 15Ga3.

Here, the replaced graphic data 15Ga1 to 15Ga3, 15Gb1 to 15Gb3, 15Gc1 to15Gc3 are stored in the form of graphic data or 3D image generatingdata.

Moreover, the target image data 15T is stored in the form of image dataobtained by shooting the object or shape data representing the featuresof the object.

FIG. 7 shows the contents of data stored in the replaced graphic datamemory (for actors) 24.

In the replaced graphic data memory (for actors) 24, each of a storagearea 24 a and a storage area 24 b stores target image data 24T the usercan register arbitrarily, replaced graphic data (1) 24Gw, marking data(1) 24Mw, replaced graphic data (2) 24Gw′, marking data (2) 24Mw′, andaudio data 24S in such a manner that the target image data 24T, thereplaced graphic data 24Gw, the marking data 24Mw, the replaced graphicdata 24Gw′, the marking data 24Mw′, and the audio data 24S correspond toone another.

In a storage area of the replaced graphic data (1) 24Gw, for example, ina storage area 24 a, a plurality of types of first-stage replacedgraphic data 24Gw1 are stored.

In a storage area of the marking data (1) 24Mw, for example, in astorage area 24 a, marking data 24Mw1 simply representing the replacedgraphic data 24Gw1 is stored.

In a storage area of the replaced graphic data (2) 24Gw′, for example,in a storage area 24 a, second-stage replaced graphic data 24Gw1′corresponding to the first-stage replaced graphic data 24Gw1 is stored.

In a storage area of the marking data (2) 24Mw′, for example, in astorage area 24 a, marking data 24Mw1′ simply representing the replacedgraphic data 24Gw1′ is stored.

In a storage area of the target image data 24T, actor's face image datato be replaced with the replaced graphic data 24Gw1 or 24Gw2 isregistered.

In a storage area of the audio data 24S, for example, in a storage area24 a, audio data 24Sw1 is stored so as to correspond to the replacedgraphic data 24Gw1 and marking data 24Mw1 and audio data 24Sw1′ isstored so as to correspond to the replaced graphic data 24Gw1′ andmarking data 24Mw1′.

When the shot image data (added marking data) Gm is displayed, or whenthe composite moving image data GG is displayed, the audio data 24S isoutput from the audio output unit 13.

The replaced graphic data 24Gw, 24Gw′ are stored in the form of graphicdata representing various motions or a combination of movement data onthe bones of the hands and feet of a person and the feature points ofthe person's face and 3D image generating data.

Although not described in detail in the embodiment, the change of thefirst-stage replaced graphic data 24Gw to the second-stage replacedgraphic data 24Gw′ in display (or composition) can be set arbitrarily.For example, the display may be changed according to the shooting dateand time or according to the date and time when the data is reproducedand displayed as composite image data.

Moreover, a plurality of face images differing in expression may beregistered in advance as image data to be recognized (on a face imagepart). Then, for example, replaced graphic data 24Gw may be read for aplacid expression and replaced graphic data 24Gw′ be read for an angryexpression and combined with the original data.

In the shot image data memory 25, a plurality of items of shot imagedata Gn taken continuously in time by the image pickup unit 21 arestored.

Sequentially Stored in the shot image data memory (added marking data)26 are the shot image data (added marking data) Gm (see FIG. 2A) createdin real time according to an image processing program for a plurality ofitems of shot image data Gn stored in the shot image data memory 25.

Stored in the composite image data memory 27 is composite image data GG(see FIG. 2B) created according to an image processing program on thebasis of the shot image data (added marking data) Gm stored in the shotimage data (added marking data) memory 26.

The digital camera 10 adds marking data to a plurality of items of shotimage data Gn taken successively by the image pickup unit 21, therebycreating shot image data (added marking data) Gm (see FIG. 2A) in realtime and displaying them. Then, the digital camera 10 creates compositeimage data GG (see FIG. 2B) from the shot image data (added markingdata) Gm recorded after shooting and displays it as a moving image oroutputs the created moving image to the outside.

Furthermore, as shown in FIG. 8, the server 30 on the network N may becaused to include the aforementioned functions of the digital camera 10.

Moreover, the server 30 may be provided with the function ofcommunicating with a digital camera 110 or a communication terminal 40via the network N. Then, the server 30 may create shot image data (addedmarking data) Gm and composite image data GG and provideresponse-delivery services to the digital camera 110 or thecommunication terminal 40.

In this case, the server 30 includes a CPU 31, a computer.

The CPU 31, which includes processing units similar to the processingunits 161 to 171 of the CPU 16 of the digital camera 10, controls theoperation of each section of the circuit according to a system programpreviously stored in a memory 32 or a server control program read intothe memory 32 via a recording medium read unit 34, such as an opticaldisk drive, from an external recording medium 33, such as a CD-ROM.

Connected to the CPU 31 are not only the memory 32 and recording mediumread unit 34 but also a communication control unit 35 for controllingdata transfer with the digital camera 110 and the communication terminal40 on the network N, an input unit 36, such as a keyboard or a mouse,and a display unit 37.

A program memory 22 of the memory 32 stores a system program whichsupervises the overall operation of the server 30, and a communicationcontrol program which supervises the operation of communicating with thedigital camera 110 and the communication terminal 40. The program memory22 further stores, in advance, an image processing program whichsupervises various functions of creating and outputting (distributing)shot image data (added marking data) Gm and composite image data GG asthe digital camera 10, on the basis of the shot image data Gntransferred from the digital camera 110 or the communication terminal40.

Then, each of the various programs stored in the program memory 22 isactivated according to an input signal from the input unit 36 or aninput signal from the digital camera 110 or the communication terminal40 via the communication control unit 35.

The memory 32 includes a changing motion data memory 14, a specialeffect graphic data memory 23, a replaced graphic data memory (forobjects) 15, a replaced graphic data memory (for actors) 24, a shotimage data memory 25, a shot image data memory (added marking data) 26,a composite image data memory 27, and a working data memory. These datamemories store almost the same data as those of the digital camera 10.

With this configuration, on the basis of a plurality of items of shotimage data Gn transferred from the digital camera 110 or thecommunication terminal 40 connected to the network N, the server 30 canriot only create similar shot image data (added marking data) Gm andcomposite image data GG but also provide response-delivery services tothe digital camera 110 and the communication terminal 40 which havetransferred the shot image data Gn.

Next, a composite image output process at the CPU 16 of the digitalcamera 10 or at the CPU 31 of the server 30 will be explained.

FIG. 9 is a flowchart to help explain all the steps of the compositeimage output process.

FIG. 10 is a sub-flowchart to help explain step SA in the flowchart ofFIG. 9, that is, the process of extracting target image data from shotimage data Gn (G1 to G9).

FIG. 11 is a sub-flowchart to help explain step SB in the flowchart ofFIG. 9, that is, a marking process A of adding marking data 24Mw to thetarget image data extracted in step SA and displaying the resultingdata.

FIG. 12 is a sub-flowchart to help explain step SC in the flowchart ofFIG. 9, that is, a marking process B of not only adding marking data15M, 15M′ to the target image data extracted in step SA and displayingthe resulting data but also outputting audio data 15S.

FIG. 13 is a sub-flowchart to help explain step SD in the flowchart ofFIG. 9, that is, a marking process C of detecting a gesture (or amotion) over a plurality of items of shot image data Gn in the targetimage data extracted in step SA and, if it has been determined that themotion is almost the same as the motion data 14P registered in thechanging motion data memory 14, adding marking data 14Mu to the targetimage data, displaying the resulting data, and outputting correspondingaudio data 14Su.

FIG. 14 is a sub-flowchart to help explain step SE in the flowchart ofFIG. 9, that is, a marking process D of detecting a gesture (or amotion) over a plurality of items of shot image data Gn in the targetimage data subjected to the marking process in step SD and, if it hasbeen determined that the motion is almost the same as a motion data 23Pregistered in the special effect graphic data memory 23, further addingmarking data 23Mu to the actor's image data, displaying the resultingdata, and outputting corresponding audio data 23Su.

FIG. 15 is a sub-flowchart to help explain step SF in the flowchart ofFIG. 9, that is, a composite process A of substituting replaced graphicdata 24Gw, 14Gu for the target image data added with marking data 24Mw,14Mu in a composite image output process to produce composite image dataand setting the output of audio data 24Sw, 14Su.

FIG. 16 is a sub-flowchart to help explain step SG in the flowchart ofFIG. 9, that is, a composite process B of substituting replaced graphicdata 15G for the target image data added with marking data 15M, 15M′ ina composite image data output process to produce a composite image andsetting the output of audio data 15S.

FIG. 17 is a sub-flowchart to help explain step SH in the flowchart ofFIG. 9, that is, a composite process C of adding replaced graphic data23Gu to a specific position of the target image data added with markingdata 23Mu in a composite image data output process to produce acomposite image and setting the output of audio data 23Su.

FIGS. 18 to 26 sequentially show the image processing states on thebasis of the shot image data Gn in the composite image output process.In FIGS. 18 to 26, A indicates shot image data Gn, B represents shotimage data (added marking data) Gm, and C shows composite image data GG.

As shown in FIGS. 18A to 26A, the processes in steps S1 to S3 of FIG. 9are executed as a result of the acquisition of shot image data Gn (G1 toG9). Then, shot image data (added marking data) Gm (Gm1 to Gm9) arecreated and output sequentially as shown in FIGS. 18B to 26B.

Furthermore, as shown in FIGS. 18B to 26B, the processes in steps S4 toS7 of FIG. 9 are performed on the shot image data (added marking data)Gm, thereby creating composite image data GG (GG1 to GG9) as shown inFIGS. 18C to 26C.

When the image pickup unit 21 of the digital camera 10 shoots a seriesof scenes shown in, for example, FIGS. 18A to 26A, a plurality of itemsof shot image data Gn (G1 to G9) are stored sequentially into the shotimage data memory 25 temporarily (step S1). Then, the sub-flowchart ofthe extracting process shown in FIG. 10 is executed (step SA).

Extracting Process

The CPU 16 temporarily transfers the shot image data Gn sequentiallystored in the shot image data memory 25 to a working memory of the CPU16 and stores them in the memory (step SA1). Then, the extracting unit161 recognizes target image data 15T, 24T for the shot image data Gn(step SA2), adds position information on the recognized target imagedata in the shot image data Gn to the shot image data Gn, and stores theresulting data into the shot image data memory 25 (step SA3).

Then, the variation between the preceding shot image data Gn-1 and thepresent shot image data Gn is blocked and extracted (step SA4). Then,position information on the image data which has the blocked variationis added to the shot image data Gn and the resulting data is stored intothe shot image data memory 25 (step SA5)

Marking Process A

After the target image data existing in the shot image data Gn isextracted in the extracting process (step SA), the first determiningunit 162 and first marking unit 163 execute the sub-flowchart of themarking process A shown in FIG. 11 (step SB).

In the marking process A, whether the target image data extracted in theextracting process has been registered in the replaced graphic datamemory (for actors) 24 as target image data 24T is determined using aknown face image recognizing process (step SB1).

If it has been determined that the target image data has been registeredas the target image data 24T (Yes in step SB1), the marking data 24Mw1in the storage area 24 a stored so as to correspond to the target imagedata 24T is read and further added to the position of the face image inthe corresponding extracted target image data. The resulting data isthen displayed and is temporarily stored into the working memory as shotimage data (added marking data) Gm1 to Gm9 as shown in FIGS. 18B to 26B(step SB2).

At this time, the marking data 24Mw1 may be displayed so as to overlapwith the face part of the extracted target image data or be near theface part.

Then, it is determined whether all of the target image data items havebeen determined. If all of them have not been determined (No in stepSB3), control returns to step SG1. If all of them have been determined(Yes in step SB3), control proceeds to the process in step SB4.

Then, if it has been determined that shooting is now in progress withthe digital camera 10 (Yes in step SB4), the audio data 24Sw1 stored inthe replaced graphic data memory (for actors) 24 is read and output fromthe audio output unit 13 (step SB5).

Next, the sub-flowchart of the marking process B shown in FIG. 12 isexecuted (step SC).

Marking Process B

In the marking process B, the second marking unit 164 determines whetherthe target image data extracted in the extracting process has beenregistered in the replaced graphic data memory (for objects) 15 astarget image data 15T (step SC1).

If in step SC1, it has been determined that the target image data hasbeen registered as target image data 15T (Yes in step SC1), the secondmarking unit 164 determines whether the extracted target image data hasbeen obtained by shooting at the best angle (step SC2). If it has beendetermined that it has been obtained by shooting at the best angle (Yesin step SC2), the marking data (before contact) 15M stored so as tocorrespond to the target image data 15T is read and added to theposition of the corresponding extracted target image data and theresulting data is displayed (step SC3).

For example, the extracted target image data corresponding to object Aand object B shown in FIGS. 18A and 19A are almost in the same state asthe target image data 15T at the best angle registered in the replacedgraphic data memory (for objects) 15.

Therefore, the marking data (before contact) 15Ma stored so as tocorrespond to the target image data 15T and the marking data (beforecontact) 15Mb are read and, at the same time, are added to the positionof the extracted target image data as shown by the shot image data Gm1,Gm2 in FIGS. 18B and 19B, respectively.

At this time, the marking data (before contact) 15Ma and marking data(before contact) 15Mb may be displayed so as to be either overlappedwith the extracted target image data or placed near the target imagedata.

Then, if it has been determined that shooting is now in progress withthe digital camera 10 (Yes in step SC4), the audio data 15S stored inthe replaced graphic data memory (for objects) 15 is read and outputfrom the audio output unit 13 (step SC5).

Thereafter, the second determining unit 165 monitors the positionalrelationship between the marking-data-added target image data and otherextracted target image data on the basis of the position informationadded to both of the target image data (step SC6). Then, the seconddetermining unit 165 determines whether the target image data eithermakes contact with or overlaps with the other extracted target imagedata (step SC7).

More specifically, in FIG. 19A, it is determined whether the targetimage data corresponding to object A makes contact with or overlaps withthe target image data corresponding to actor P. In FIG. 20A, it isdetermined whether the target image data corresponding to object B makescontact with or overlaps with the target image data corresponding toactor P.

If it has been determined that position information on the target imagedata makes contact with or overlaps with position information on theother extracted target image data (Yes in step SC7), it is determinedwhether or not the other extracted target image data is a part of thetarget image data to which the marking data 24M has been added in themarking process A (step SC8).

More specifically, for in FIG. 19A, it is determined whether the markingdata (1) 24Mw1 has been added to the target image data corresponding toactor P.

If in step SC8, it has been determined that it is a part of the otherextracted target image data to which the marking data 24MW has beenadded (Yes in step SC8), the third making unit 166 reads marking data(after contact) 15M′, substitutes the marking data 15M′ for the markingdata (before contact) 15M added in step SC3, stores the resulting datainto the working data memory temporarily, and displays the shot imagedata (added marking data) Gm (step SC9).

The process in step SC9 will be explained using FIG. 19B. The markingdata (after contact) 15Ma′ stored so as to correspond to the targetimage data 15T is read and added to the position of the marking data(before contact) 15Ma, and substituted for the marking data (beforecontact) 15Ma, and the resulting data is stored into the working datamemory temporarily, and the data is displayed on the display unit 11(step SC9).

Then, if it has been determined that shooting is now in progress withthe digital camera 10 (Yes in step SC10), the audio data 15S stored soas to correspond to the target image data 15T is read and output fromthe audio output unit 13 (step SC11).

In step SC2, if it has been determined that the extracted target imagedata has not been picked up at the best angle (No in step SC2), themarking data (after contact) 15M′ stored so as to correspond to thetarget image data 15T is read and added to the position of thecorresponding extracted target image data and the resulting data isdisplayed (step SC12).

Then, if it has been determined that shooting is now in progress withthe digital camera 10 (Yes in step SC13), the audio data 15S stored soas to correspond to the target image data 15T is read and output fromthe audio output unit 13 (step SC14).

Next, the sub-flowchart of the marking process C shown in FIG. 13 isexecuted (step SD).

Marking Process C

In the marking process C, the third determining unit 167 determineswhether the target image data extracted in the extracting process hasmoved over a plurality of items of shot image data Gn and its movement(or motion) is almost the same as the motion data 14P previously storedin the changing motion data memory 14 (step SD1).

Step SD1 will be explained using FIGS. 21 to 23.

As shown in FIGS. 21A to 23A, it is determined whether the motion of thetarget image data corresponding to actor K is almost the same as themotion data 14P (14P1 a to 14P1 c) previously stored in the changingmotion data memory 14.

If it has been determined that the motion is almost the same as themotion data 14P (Yes in step SD1), the fourth marking unit 168 reads themarking data 14Mu stored so as to correspond to the motion data 14P,adds the marking data 14Mu to the image of the face part of theextracted target image data, displays the resulting data, and stores thedata into the working memory temporarily as shot image data (addedmarking data) Gm6 to Gm9 as shown in FIGS. 23B to 26B (step SD2).

At this time, the marking data 14Mu1 may be displayed so as to eitheroverlap with or be placed near the face part of the extracted targetimage data.

Then, if it has been determined that shooting is now in progress withthe digital camera 10 (Yes in step SD3), the audio data 14Su stored soas to correspond to the motion data 14P is read and output from theaudio output unit 13 (step SD4).

Next, the sub-flowchart of the marking process D shown in FIG. 14 isexecuted (step SE).

Marking Process D

In the marking process D, the third determining unit 167 determineswhether the target image data to which making data has been added in themarking process C has moved over a plurality of items of shot image dataGn and the movement (or motion) of the target image data is almost thesame as the motion data previously stored in the special effect graphicdata memory 23 (step SE1).

Step SE1 will be explained using FIGS. 24 to 26.

As shown in FIGS. 24B to 26B, it is determined whether the motion of thetarget image data to which the marking data 14Mu has been added isalmost the same as the motion data 23P (23P1 a to 23P1 c) previouslystored in the special effect graphic data memory 23.

If it has been determined that the motion is almost the same as themotion data 23P (Yes in step SE1), the fourth marking unit 168 reads themarking data 23Mu1 stored so as to correspond to the motion data 23P,adds the mark data 23Mu1 to the image of the face part of the extractedtarget image data, displays the resulting data, and stores the data intothe working data memory temporarily as shot image data (added markingdata) Gm9 as shown in FIG. 26B (step SE2).

Then, if it has been determined that shooting is now in progress withthe digital camera 10 (Yes in step SE3), the audio data 23Su stored soas to correspond to the motion data 23P is read and output from theaudio output unit 13 (step SE4).

Thereafter, control proceeds to the process in step S2.

Here, the marking data 23Mu1 may be displayed so as to either overlapwith or be placed near the face part of the extracted target image data.

The shot image data (added marking data) Gm1 to Gm9 consecutivelycreated on the working memory in the processes of step SA to step SE arestored into the added mark data to shot image data memory 26 (step S2).

If it has been determined that the shooting of a series of shot imagedata Gn (G1 to G9) with the image pickup unit 21 has been completed (Yesin step S3), it is determined whether control proceeds to the process ofgenerating a composite image on the basis of a series of shot image data(added marking data) Gm (Gm1 to Gm9) stored in the shot image data(added marking data) memory 26 (step S4).

If the generation of a composite image is specified as a result of thedetection of the input of an operation signal from the input unit 12 orif the acquisition of shot image data and the completion of the markingprocess have been detected, the CPU 16 determines that control proceedsto the process of generating a composite image (Yes in step S5).

Then, the shot image data (added marking data) Gm (Gm1 to Gm9) is readfrom the shot image data (added marking data) memory 26 into the workingmemory and control proceeds to the composite process A of FIG. 15 (stepSF).

Composite Process A

The CPU 16 determines whether the shot image data (added marking data)Gm (Gm1 to Gm9) read from the shot image data (added marking data)memory 26 into the working memory include target image data to whichmarking data has been added in the marking processes A and C (step SF1).

Step SF1 will be explained using FIGS. 18B to 26B. It is determinedwhether target image data (image data on actor P) to which marking data24Mw1 has been added or target image data (image data on actor K) towhich marking data 14Mu1 has been added is present in the shot imagedata (added marking data) Gm.

Then, if it has been determined that target image data is present (Yesin step SF1), the moving image generating unit 170 captures the movementof target image data in the shot image data Gm (Gm1 to Gm9) on the basisof position information added to the target image data to which themarking data has been added (step SF2).

Then, the replacing unit 169 reads the replaced graphic data (1) 24Gw1corresponding to the marking data 24Mw1 in the replaced graphic datamemory (for actors) 24 and the replaced graphic data 14Gu1 correspondingto the marking data 14Mu1 in the changing motion data memory 14 anddraws replaced graphic data in a plurality of poses shown in FIGS. 18Cto 26C according to the captured movement.

Thereafter, the composite moving image generating unit 170 replaces thereplaced graphic data on the basis of position information on thecorresponding target image data and creates composite image data GG (GG1to GG9) combined with a prepared background image GG (step SF3).

At this time, as shown by arrow x in FIG. 23C, replaced graphic data14Gu1 that draws the same pose as that of actor K is enlarged at thetime of drawing.

Then, the CPU 16 reads the audio data 24Sw1 stored so as to correspondto the replaced graphic data 24Gw1 and stores the audio data 24Sw1 insuch a manner that it corresponds to composite images GG (GG1 to GG9).

Moreover, the CPU 16 reads the audio data 14Su1 stored so as tocorrespond to the replaced graphic data 14Gu1 and stores audio data14Su1 in such a manner that it corresponds to composite images GG (GG6to GG9) (step SF4).

Next, the sub-flowchart of the composite process B shown in FIG. 16 isexecuted (step SG).

Composite Process B

The CPU 16 determines whether the shot image data (added marking data)Gm (Gm1 to Gm9) read from the shot image data (added marking data)memory 26 into the working memory include target image data to whichmarking data has been added in the marking process B (step SG1).

Step SG1 will be explained using FIGS. 18B to 26B. It is determinedwhether target image data (image data on object A) to which marking data15Ma1 has been added or target image data (image data on object B) towhich marking data 15Mb1 has been added is present in the shot imagedata (added marking data) Gm.

Then, if it has been determined that target image data is present (Yesin step SG1), the composite moving image generating unit 170 reads thereplaced graphic data corresponding to the marking data in the replacedgraphic data memory (for objects) 15, replaces the replaced graphic dataon the basis of position information on the corresponding target imagedata, and adds the resulting data to the composite moving image data GG(GG1 to GG9) created in step SF3 to produce composite image data (stepSG2).

Step SG2 will be explained using FIGS. 18B to 26B. The target image data(image data on object B) to which marking data (before contact) 15Mb1has been added is replaced with replaced graphic data 15Gb1. Theresulting data is added to the composite moving image data created instep SF3 to produce composite image data.

Then, the CPU 16 reads the audio data 15Sb1 stored so as to correspondto the replaced graphic data 15 b 1 and stores the audio data 15Sb1 insuch a manner that it corresponds to the composite images GG (GG1 toGG9) (step SG3).

Next, the sub-flowchart of the composite process C shown in FIG. 17 isexecuted (step SH).

Composite Process C

The CPU 16 determines whether the shot image data (added marking data)Gm (Gm1 to Gm9) read from the shot image data memory (added markingdata) 26 into the working memory include target image data to whichmarking data has been added in the marking process D (step SH1).

Step SH1 will be explained using FIG. 26B. It is determined whethertarget image data to which marking data 23Mu1 has been added is presentin the shot image data (added marking data) Gm.

Then, if it has been determined that target image data is present (Yesin step SH1), the composite moving image generating unit 170 reads thereplaced graphic data corresponding to the marking data in the specialeffect graphic data memory 23, replaces the replaced graphic data on thebasis of position information on the corresponding marking data, andadds the resulting data to the composite moving image data GG (GG1 toGG9) created in step SG2 to produce composite image data (step SH2).

Furthermore, the composite moving image generating unit 170 acquiresposition information on the composite replaced graphic data, anddetermines the positional relationship between the peripheral part ofthe replaced graphic data and other replaced graphic data. If theperipheral part of the replaced graphic data makes contact with oroverlaps with the other replaced graphic data, the composite movingimage generating unit 170 further adds position information on thecontact or overlap to produce composite image data (step SH3).

Step SH2 and step SF3 will be explained using FIG. 26C. Replaced graphicdata 23Gu1 is added to the position to which marking data 23Mu1 has beenadded to produce composite image data. Moreover, graphic data 23Gu1′ isadded to the position where the replaced graphic data 23Gu1 makescontact with replaced graphic data 24Gw1 to produce composite imagedata.

Then, the CPU 16 reads the audio data 23Su1 stored so as to correspondto the replaced graphic data 23Gu1 and stores it in such a manner thatthe audio data 23Su1 corresponds to the composite images GG (GG1 to GG9)(step SH4).

Thereafter, the CPU 16 converts the composite images GG (GG1 to GG9)produced in the composite processes A to C into audio-data-addedcomposite image data as shown in FIG. 9 and stores the resulting datainto the composite image data memory 27 (step S6).

Then, the composite moving image data is output to the display unit 11,which reproduces and displays the data (step S7).

When carrying out the above-described processes, the server 30 reads thecomposite moving image data stored in the composite image data memory 27and transfers the data to the digital camera or the communicationterminal 40, the sender of the shot image data Gn (G1 to G9) received instep S1 (step S7).

Accordingly, it is possible to create a composite moving image easilyaccording to the actor's will.

The individual processes performed on the composite image output unitdescribed in the embodiment can be stored in the form of a programexecutable on a computer into an external recording medium (33), such asa memory card (e.g., a ROM card or a RAM card), a magnetic disk (e.g., afloppy disk or a hard disk), an optical disk (e.g., CD-ROM or a DVD), ora semiconductor memory. Then, the external storage medium can bedistributed.

Furthermore, program data to realize the individual methods can betransferred in the form of program code on a network (e.g., theInternet) N. The program data can be taken in from a computer terminal(e.g., a program server) connected to the network (or the Internet) Nand the function of creating composite moving image data from theaforementioned shot image data Gn and outputting the resulting data canbe realized.

Additional advantages and modifications will readily occur to thoseskilled in the art. Therefore, the invention in its broader aspects isnot limited to the specific details and representative embodiments shownand described herein. Accordingly, various modifications may be madewithout departing from the spirit or scope of the general inventiveconcept as defined by the appended claims and their equivalents.

1. A composite image generating apparatus comprising: a first storageunit configured to store a motion to be combined and first compositeimage data in such a manner that the motion and the first compositeimage data correspond to each other; an input unit configured to input aplurality of items of image data; a first determining unit configured todetermine whether or not the movement of a part changing over aplurality of items of image data input by the input unit is almost thesame as the motion stored in the first storage unit; and a firstcomposite unit configured to read the first composite image data storedin the first storage unit so as to correspond to the motion and combinethe first composite data with the image data having the changing part,if the first determining unit determines that the movement is almost thesame as the motion.
 2. The composite image generating apparatusaccording to claim 1, further comprising: a second storage unitconfigured to store image data to be extracted; a second determiningunit configured to determine whether or not the image data input by theinput unit includes the image data to be extracted stored in the secondstorage unit; and a second composite unit configured to add specificimage data to the composite image data produced by the first compositeunit on basis of the positional relationship between the image data tobe extracted and image data with the changing part to produce compositeimage data, if the second determining unit determines that the inputimage data includes the image data to be extracted.
 3. The compositeimage generating apparatus according to claim 2, wherein the secondstorage unit configured to further store second composite image data insuch a manner that the second composite image data and the image data tohe extracted correspond to each other, and the composite imagegenerating apparatus further comprises third composite unit configuredto read the second composite image data stored so as to correspond tothe image data to be extracted and add the second composite image datato the position of the image data to be extracted to produce compositeimage data, if the second determining unit determines that the inputimage data includes the image data to be extracted.
 4. The compositeimage generating apparatus according to claim 3, further comprising: athird storage unit configured to store a motion of image data having thechanging part and third composite image data in such a manner that themotion and the third composite image data corresponds to each other; athird determining unit configured to determine whether or not themovement of the image data having the changing part over a plurality ofitems of image data input by the input unit is almost the same as themotion stored in the third storage unit; and a fourth composite unitconfigured to read the third composite image data stored in the thirdstorage unit so as to correspond to the motion and add the thirdcomposite image data to the composite image data produced by the firstcomposite unit to produce composite image data, if the third determiningunit determines that the movement is almost the same as the motion. 5.The composite image generating apparatus according to claim 1, furthercomprising: a moving image data creating unit configured to createmoving image data by changing the first composite image data inaccordance with the movement of image data in the part changing over aplurality of items of image data, wherein the first composite unitcreates composite moving image data which includes moving data createdby the moving image data creating unit.
 6. The composite imagegenerating apparatus according to claim 1, wherein the input unitincludes image shooting unit.
 7. The composite image generatingapparatus according to claim 1, wherein the input unit includescommunication unit configured to input said plurality of items of imagedata in communication with the outside.
 8. The composite imagegenerating apparatus according to claim 1, further comprising: a displayunit configured to display a composite image data produced by the firstcomposite unit.
 9. The composite image generating apparatus according toclaim 1, wherein the first composite image data is 3D images.
 10. Acomposite image generating method comprising: an input step of inputtinga plurality of items of image data; a determination step of determiningwhether or not the movement of a part changing over a plurality of itemsof image data input in the input step is almost the same as a motion setso as to correspond to composite image data; and a composite step of, ifit is determined in the determination step that the movement is almostthe same as the motion, combining the composite image data set so as tocorrespond to the motion with image data having the changing part.
 11. Acomputer readable medium for storing a program product for use withimage generating apparatus including an image input unit, the programproduct comprising: first computer readable program means for inputtinga plurality of items of image data by the image input unit, secondcomputer readable program means for determining whether or not themovement of a part changing over a plurality of items of image datainput by the first computer readable program means is almost the same asa motion set so as to correspond to composite image data, and thirdcomputer readable program means for, if the second computer readableprogram means determines that the movement is almost the same as themotion, combining the composite image data set so as to correspond tothe motion with the image data having the changing part.